Conditional Mutual Information Neural Estimator

Several recent works in communication systems have proposed to leverage the power of neural networks in the design of encoders and decoders. In this approach, these blocks can be tailored to maximize the transmission rate based on aggregated samples from the channel. Motivated by the fact that, in many communication schemes, the achievable transmission rate is determined by a conditional mutual information term, this paper focuses on neural-based estimators for this information-theoretic quantity. Our results are based on variational bounds for the KL-divergence and, in contrast to some previous works, we provide a mathematically rigorous lower bound. However, additional challenges with respect to the unconditional mutual information emerge due to the presence of a conditional density function which we address here.Comment: To be presented at ICASSP 202

Neural Estimator of Information for Time-Series Data with Dependency

Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff’s ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations

Statistical Inference of Information in Networks : Causality and Directed Information Graphs

Over the last decades, the advancements in measurement, collection, and storage of data have provided tremendous amounts of information. Thus, it has become crucial to extract valuable features and analyze the characteristics of data. As we study more complex systems (e.g. a network of sensors), the relationship between the information in different parts (e.g. measured signals) brings more insight in describing the characteristics of the system. Quantities such as entropy, mutual information, and directed information (DI) can be employed for this purpose. The main theme of this thesis is to study causality between random processes in systems where the instantaneous samples may depend on the history of other processes. We justify utilizing DI to describe the extent of causal influence and provide appropriate tools to estimate this quantity. Additionally, we study properties of the directed information graph, a representation model to demonstrate causal relationships in a network of processes. Although conventional estimation techniques for information-theoretic quantities may suit small systems with low-dimensional data, recent studies acknowledge that these methods may encounter a deterioration in performance when the data is high-dimensional. The estimation techniques proposed in this thesis are aimed to tackle this issue by using a novel approach based on neural networks. A major challenge of this method to estimate DI is to construct appropriate data batches to train the neural network. Thus, we propose a technique using the $k$ nearest neighbors ($k$-NN) algorithm to re-sample the original data. Since DI is characterized with conditional mutual information (CMI) terms, the convergence of our estimators is shown in two steps. First, we prove that the estimation for CMI converges asymptotically to the true value, when samples are independent and identically distributed (i.i.d.). The proof includes a concentration bound for our $k$-NN re-sampling technique. In the next step, the results are extended to the case where samples are allowed to be dependent in time which enables the method to estimate DI. Accordingly, we provide the convergence results for the end-to-end estimation of DI in this scenario. The performance of estimations is investigated in several experiments both with synthetic and real-world data.  In order to detect a causal link in the system, a threshold test can be performed on the estimated DI. However, for more complex systems, where the directed information graph representation is adopted, it is required to detect all causal links correctly. Therefore, we study the performance of detecting the whole graph and show that with the conventional empirical estimation method and properly choosing the threshold for the test, the type I and II error probabilities tend to zero asymptotically. Finally, we suggest a roadmap to extend these results for a finite-sample regime to obtain explicit bounds describing the behavior of the false alarm and detection probabilities. QC 20211001</p

Statistical Inference of Information in Networks : Causality and Directed Information Graphs

Over the last decades, the advancements in measurement, collection, and storage of data have provided tremendous amounts of information. Thus, it has become crucial to extract valuable features and analyze the characteristics of data. As we study more complex systems (e.g. a network of sensors), the relationship between the information in different parts (e.g. measured signals) brings more insight in describing the characteristics of the system. Quantities such as entropy, mutual information, and directed information (DI) can be employed for this purpose. The main theme of this thesis is to study causality between random processes in systems where the instantaneous samples may depend on the history of other processes. We justify utilizing DI to describe the extent of causal influence and provide appropriate tools to estimate this quantity. Additionally, we study properties of the directed information graph, a representation model to demonstrate causal relationships in a network of processes. Although conventional estimation techniques for information-theoretic quantities may suit small systems with low-dimensional data, recent studies acknowledge that these methods may encounter a deterioration in performance when the data is high-dimensional. The estimation techniques proposed in this thesis are aimed to tackle this issue by using a novel approach based on neural networks. A major challenge of this method to estimate DI is to construct appropriate data batches to train the neural network. Thus, we propose a technique using the $k$ nearest neighbors ($k$-NN) algorithm to re-sample the original data. Since DI is characterized with conditional mutual information (CMI) terms, the convergence of our estimators is shown in two steps. First, we prove that the estimation for CMI converges asymptotically to the true value, when samples are independent and identically distributed (i.i.d.). The proof includes a concentration bound for our $k$-NN re-sampling technique. In the next step, the results are extended to the case where samples are allowed to be dependent in time which enables the method to estimate DI. Accordingly, we provide the convergence results for the end-to-end estimation of DI in this scenario. The performance of estimations is investigated in several experiments both with synthetic and real-world data.  In order to detect a causal link in the system, a threshold test can be performed on the estimated DI. However, for more complex systems, where the directed information graph representation is adopted, it is required to detect all causal links correctly. Therefore, we study the performance of detecting the whole graph and show that with the conventional empirical estimation method and properly choosing the threshold for the test, the type I and II error probabilities tend to zero asymptotically. Finally, we suggest a roadmap to extend these results for a finite-sample regime to obtain explicit bounds describing the behavior of the false alarm and detection probabilities. QC 20211001</p

Testing for Directed Information Graphs

In this paper, we study a hypothesis test to determine the underlying directed graph structure of nodes in a network, where the nodes represent random processes and the direction of the links indicate a causal relationship between said processes. Specifically, a k-th order Markov structure is considered for them, and the chosen metric to determine a connection between nodes is the directed information. The hypothesis test is based on the empirically calculated transition probabilities which are used to estimate the directed information. For a single edge, it is proven that the detection probability can be chosen arbitrarily close to one, while the false alarm probability remains negligible. When the test is performed on the whole graph, we derive bounds for the false alarm and detection probabilities, which show that the test is asymptotically optimal by properly setting the threshold test and using a large number of samples. Furthermore, we study how the convergence of the measures relies on the existence of links in the true graph.QC 20180419</p